@chengzhuzhang We had discussed determining how many copies of the simulation data we had, and if any of them could be removed. The script added here checks a number of paths we might expect to find data. The results:

Things to note:

1. Several simulations now have copies both in the central location and in the original user path (sometimes even 2 users' paths!). So, we can clear some space by telling those users they're free to remove those simulations.
2. In all cases where there is data in the publication archive, the central location already simply symlinks to it (thus not actually duplicating any data).
3. There's still the question of file ownership for the data physically present in the centralized location. I'm currently the owner of all those. So to ensure I'm within space quotas, we're going to need to a) transfer those directories to someone else, or b) increase my quota. @ndkeen -- could you advise on that?

@forsyth2 It's good to have this information, but at this point I don't think we need to ask user to delete data. Based on the table, it does look like the v1 BGC simulations are not included, which you can use Tony's copy under publication-archive

at this point I don't think we need to ask user to delete data.

Well, NERSC is generally adverse to having duplicated data, but if we have the space as a project, I suppose it's ok. But again, there's still the problem that I do not have the space as an individual user of NERSC.

does look like the v1 BGC simulations are not included,

Yes, that's correct. They previously weren't included on the list of simulations to include and I haven't had a chance to add them yet. On that note, v1 CRYO (low-resolution) is the only other remaining dataset listed on https://e3sm.atlassian.net/wiki/spaces/ED/pages/4495441922/V1+Simulation+backfill+WIP? Should I add both BGC and cryo, or just BGC?

at this point I don't think we need to ask user to delete data.

Well, NERSC is generally adverse to having duplicated data, but if we have the space as a project, I suppose it's ok. But again, there's still the problem that I do not have the space as an individual user of NERSC.

Let's hear from @ndkeen how to best resolve this space problem.

does look like the v1 BGC simulations are not included,

Yes, that's correct. They previously weren't included on the list of simulations to include and I haven't had a chance to add them yet. On that note, v1 CRYO (low-resolution) is the only other remaining dataset listed on https://e3sm.atlassian.net/wiki/spaces/ED/pages/4495441922/V1+Simulation+backfill+WIP? Should I add both BGC and cryo, or just BGC?

Good catch! Yes, let's add both BGC and cryo to make it complete

This PR is for this script that counts data duplicates, so we can move discussion of remaining v1 data to #63.